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FOREWORD 


The three volumes of this final report present the results of a 
study to develop a program for investigating reliability aspects of the 
Ground Support Equipment of the Kennedy Space Center (KSC). The 
study is divided into two distinct parts: (1) investigating the reliability 
of equipment already in use, including field failure rate generation, 
and (2) investigating the reliability of equipment not yet in design, with 
particular reference to hardware /software configurations of large real- 
time systems. The work was performed by Planning Research Corpo- 
ration under Contract Number NAS10-7621 during the period 24 May 
1971 through 24 May 1972 for the Systems Engineering Division, De- 
sign Engineering Directorate, John F. Kennedy Space Center. Mr. R. E. 
Cato, Jr. and Mr. R. Galloway were the technical monitors of various 
portions of the study, in cooperation with Mr. Otto Fedor. 

A study of this type involves the contributions of a number of people. 
The forward to each volume identifies the PRC personnel responsible 
for the work reported within that volume. Major authors of each vol- 
ume are identified on its fly leaf. 

Ms. E. E. Bean was the PRC Project Manager for the entire study. 
E. E. Bean and C. E. Bloomquist were the study team members respon- 
sible for Volume I. 
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ABSTRACT 


The three volumes of this report document a variety of activities 
undertaken in the development of a KSC program for investigating and 
generating field failure rates for the Ground Support Equipment (GSE) 
designed by KSC. 

The first volume contains a summary and overview of the work 
accomplished during this study and, as such, summarizes Volumes II 
and III. Volume II is in the form of a handbook that KSC can use for 
dissemination of field reliability data generated under this and antece- 
dent contracts. KSC can add data to the handbook using the methodology 
contained therein for operating on the Unsatisfactory Condition Reports 
(UCRs) available at KSC. The handbook format has been designed spe- 
cifically for this purpose and is completely self-contained. It includes 
summary data, 20 complete Reliability Assessments of Components 
(RACs), and step-by-step procedures for generating additional data. 
Issuance of this handbook as a KSC working document requires only 
the removal of the PRC cover page and forward. Volume III collects 
the work performed in this study on the problem of how to obtain reli- 
able (i. e., error free) software when software is considered to be a 
component of a hardware/ software ground support system operating in 
real time. Included in Volume III are recommended procedures to be 
employed in various phases of software production, from the early de- 
sign stage to the sustaining engineering phase. A data collection sys- 
tem for the software component of the hardware /software configuration 
analogous to the UCR system is also recommended together with an in- 
dication of the analysis and use of such data to manage costs and sched- 
ules associated with software production. 
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I. INTRODUCTION 


This final report, published in three volumes, documents the 
work performed for the Kennedy Space Center (KSC) in establishing a 
KSC program for the continuing investigation of the reliability of Ground 
Support Equipment designed by KSC. This volume. Volume I, presents 
a summary and overview of the completed study tasks. Volume II con- 
tains a recommended handbook format for displaying ground support 
equipment (GSE) reliability characteristics, illustrated with data from 
the Reliability Assessment of Components (RACs) generated during this 
study. Volume III contains the recommended procedures for attaining 
reliable (i.e., error-free) software used as a ground system component 
in hardware /software configurations operating in real time. 

A. Background 

The work documented in this report is a continuation of a study 
performed for KSC over the past several years. Earlier effort had 
addressed the question: Can technician- and engineer-recorded field 
information be utilized profitably in a reliability assessment of compo- 
nents in field usage? Specifically, can such an assessment be obtained 
by analyzing solely the data contained in the then defined Unsatisfactory 
Condition Report (UCR) historical file retained at KSC? The analysis 
of the UCR system and the development of a methodology to extract 
pertinent and useful reliability information was reported in Reference 1. 
The methodology as contained in Reference 1 was applied to four 
mechanical/electromechanical components 1 of the GSE: Tail Service 
Masts, Umbilical Swing Arms, regulators and solenoid valves. 

The results of Reference 1 were encouraging enough to proceed 
with the second phase of the study; that is, to determine if the method- 
ology developed could also be applied to electrical and electronic 

throughout the discussions of this report, "component" is defined as a 
matrix of parts, assemblies, or subassemblies, typically self-contained, 
that function in a defined manner relative to overall equipment operation. 
Defined in this manner equipments of widely different complexity may be 
termed 1 components, " Tail Service Masts and regulators, for example. 
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components and, if so, to develop a reporting document for the relia- 
bility assessment of components (RACs) of KSC GSE that could be used 
by various KSC personnel. Reference 2 documents the results of that 
activity. 

Reference 2 contains the final version of the developed methodology 
and RACs for seven GSE components. The recommended method of dis- 
semination of the information of the RACs in Reference 2 was by incor- 
poration into the handbook of the Kennedy Approved Parts List, a docu- 
mentation system then being developed concurrently with the Phase II 
study. 

The study documented in the three volumes of this report is the 
third phase in the overall effort even though the reports are not officially 
titled Phase III. The original study plan for this phase called for three 
specific tasks. The first was to review the UCR system as it operated 
in early 1971 (the system underwent radical changes in October 1969) 
and to determine the impact, if any, on the methodology developed, 
under the earlier system. The second task involved continued method- 
ology application. PRC's effort in this task was two-fold. First, the 
RACs for 11 critical components as defined by KSC were to be generated 
by PRC personnel and a format for a potential KSC handbook for dis- 
playing the reliability information was to be developed. Second, PRC 
personnel were to act as consultants in training sessions of KSC per- 
sonnel to enable a smooth "transfer" of the techniques involved in RAC 
generation. The third original task was defined as methodology exten- 
sion incorporating new areas, such as testing and maintenance. 

Subsequent to the beginning of the study activities, the third task 
discussed above was eliminated and replaced with the task concerning 
reliability of hardware /software configurations used in real-time, 
ground support systems. 

B. Summary of Study Tasks 

1. Task 1; Review of UCR System 

The major objective of this task was to review the UCR 
system as it currently operates with particular reference to the impact. 
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if any, on the methodology for RAC production developed prior to the 
date of the UCR revision (15 October 1969). The review indicated that 
the methodology is applicable to both the revised system and the system 
as it existed prior to 15 October 1969. Several recommendations con- 
cerning both the UCR form, the UCR coding system, and the overall 
mechanics of the system are made in Section II as a result of this review. 

2. Task 2; Methodology Application 

This task had two major objectives. The first was to pro- 
vide "transference assistance" to KSC personnel in the effort to have 
KSC personnel assume all RAC production activities. Conditions pre- 
vailing at KSC soon after the beginning of this study limited this train- 
ing activity to KSC supervisory personnel and the writing of a document 
giving step-by-step procedures (i. e., a training manual) for RAC 
generation. 

The second major objective of this task was to generate RACs 
for a KSC selected list of components. The 15 RACs generated by PRC 
personnel are identified in Section III of this volume and are published 
in Volume II of this report. 

3. Reliability of Hardware /Software Configurations Used in 

Real-Time, Ground Support Applications 

The objective of this task was to identify factors that affect 
the reliability of hardware/ software configurations for large, complex, 
real-time computer systems. Three major areas were addressed: 

(1) documentation of the collective effort to date in the computer industry 
that is directed toward improving the reliability or quality of such con- 
figurations; (2) identification of factors that affect the combined relia- 
bility of such a configuration; and (3) development of criteria and/or 
guidelines useful to KSC in its effort to develop and operate such a con- 
figuration within limited funds and under severe time constraints. The 
results are documented in Volume III of this report. 

Volume III shows that the current state-of-the-art of software 
reliability is limited to steps that can be taken to "build reliability into" 
a hardware /software configuration. While it is desirable to measure 
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software reliability or effectiveness as is done in hardware, this ability 
must await further theoretical effort. However, as Volume III points 
out, there are measures that can be made of a hardware /software con- 
figuration in the development process that will aid in control of the 
software reliability. 

C. Organization of Volume I 

Section II of this document contains the review of the present (1972) 
UCR system with recommendations for its improvement. Section III dis- 
cusses the RAC generation performed by PRC in this study, addresses 
topics pertinent to its application by KSC personnel, and briefly describes 
the recommended format for a KSC handbook on reliability of GSE. 

Section IV summarizes the work performed in the hardware /software 
reliability task. Text references follow Section IV. The Bibliography 
lists all documentation supplied by KSC as source data for this study. 
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II. REVIEW OF UCR SYSTEM 


Earlier phases of the overall effort investigating the reliability 
of the KSC Ground Support Equipment were based on the UCR system 
that was in effect until 15 October 1969. Subsequent to that date several 
changes were made in the system, both in terms of actual data collected 
and of physical methods used to store and retrieve the UCR information. 
This section considers the "new" UCR system (i. e., the one in effect 
after 15 October 1969) by briefly discussing certain aspects of the re- 
vised system, by assessing its impact on RAC generation, and finally 
by providing recommendations for KSC consideration for improvement 
of the UCR system. 

A. The UCR System After 15 October 1969 

1. General 

The UCR system as it existed prior to 15 October 1969 was 
described and critiqued in Reference 1. Knowledge of the material 
given in Reference 1 is presupposed in the discussion to follow. 

In the summer of 1969 the reporting format for UCRs was revised 
to produce a somewhat simpler form. For convenience of discussion, 
the revised form is shown in Exhibit 1. 1 In general, the instructions 
for its generation did not vary significantly from the instructions pre- 
viously used. The purpose for the form was retained: to report to 
Design Engineering (DE) the unsatisfactory condition of any element 
of the GSE under the cognizance of DE for which action by DE is 
required. 

Even though the expressed purpose did not change before and 
after 15 October 1969, as a practical matter significant changes did 
occur. The most obvious, and most dramatic, change is the reduced 
rate of UCR submittals. Several reasons have been postulated for this 
reduction: 


Exhibit 1 is a reproduction of a UCR submitted on the revised form. 
The content of the UCR is not pertinent to the discussion. The type of 
information requested in each block of the form is pertinent. 





EXHIBIT 1 - REVISED UCR FORM 
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o Many previous UCRs were submitted that "should not have 
been" i. e. , many purely maintenance actions were being 
erroneously reported on UCRs rather than on Discrepancy 
Reports (DR 1 s) 

o The current system incorporates a screening process, thus 
eliminating inappropriate UCRs from the data bank 
o The reduction is due to system stability, i. e. , reliability 
growth 

o The reduction is due to decreasing launch frequency 

o The reduction is due to a combination of the above. 

Brief comments on each of the above reasons are in order. 

The analysis performed in this study does not support the propo- 
sition that different types of problems are being reported in the new 
system. That is, maintenance type actions are still being reported. 

The second postulated reason for the decreased rate of UCR gen- 
eration--the screening process--is a contributing factor to the decreased 
rate but only a very small one. Investigation showed that of all UCRs 
submitted to the data bank since 15 October 1969 only 43 had been re- 
jected (less than 5 percent of the total). 

There is no evidence in the analyses performed to confirm or 
deny that system stability or reliability growth is a contributing factor 
to the decreased rate of submittal. If a significant increase in system 
stability had occurred, it seems reasonable to expect to see it reflected 
in the new data. The data sample from the new system for each RAC 
is as yet too small, however, to make a statistically valid statement 
one way or the other. The only observation that can be made is that 
field failure rates based on the new data are not all lower than those 
computed on the data base of the old system. 

PRC submits that if system stability is indeed a contributing factor, 
DE should be able to readily observe this phenomenon via other means. 
For example, a significant decrease in downtime and/or repair activi- 
ties and a significant reduction in cost of supporting the last few launches 
should be easily discernible. Complete resolution of this point was be- 
yond the scope of this study. 
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The last point cited above- -decreasing launch frequency- -does 
not appear to be valid. Exhibit 2 shows the number of UCRs in both 
the old and new system by calendar quarter with vehicle launch dates 
superimposed. Examination of the exhibit clearly indicates that the de- 
cline in the rate of UCR generation actually began in the first quarter 
of 1968 and has continued since that time. Note, however, that from 
the last quarter of 1968 to the end of 1969 launch frequency was not de- 
clining (six launches in 14 months) while UCRs submitted declined 
steadily from approximately 1,800 per quarter to approximately 200 per 
quarter. 

The actual reason for the rapidly declining rate of UCR submittal 
is therefore basically unknown to the study team. It most likely involves 
some combination of the reasons listed above and perhaps some others 
which are as yet unknown. Emphasis by DE to restrict use of the UCR 
form to purposes for which it was originally intended may well have 
caused UCR originators to reduce their output. There also was, in fact, 
a decrease in launch frequency after 1969, and an increase in reliability 
for at least some components of the GSE is not unlikely. In view of 
KSC's intent to continue generating RACs it is to be hoped that DRs 
are in fact being generated for field problems of interest to this 
activity. 

Other changes evident in the new system are related to entries on 
the revised form of Exhibit 1 and to UCR coding for file storage. Each 
of these is discussed in the following paragraphs. 

2. Comments on Entries of Revised UCR Form 

The revised form shown in Exhibit 1 does not include certain 
data elements needed for studies of this type. The omitted data elements 
were included in the old system. In PRC' s opinion the most serious 
omissions are (1) part serial number and (2) replacement part serial 
number. If the data element required by block 12 on Exhibit 1 (installed 
time) were always provided, the omission of the serial numbers might 
not be as significant as it currently is. The developed methodology 
contains a method of obtaining time information through use of serial 



Vehicle Launch Date 
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numbers, a method which cannot be utilized under the new system. 

Time is the most often missing item in the old system as well as the 
new. Of the 963 UCRs of the new system available to this study, only 
274 have entries in block 12. 

Equally as serious an omission is the next assembly part number 
and serial number. There are components for which a RAC is desired 
but for which there is no data retrieval method except by next assembly 
part number (Tail Service Masts; at Launch Complex 39, for example). 
For this study analysis, the Tail Service Mast RAC was updated, but 
only because of the small number of UCRs in the new system («1,000) 
was amenable to hand sorting. Any growth in the UCR historical file 
would seriously hamper the ability to retrieve such components manually. 

In the analyses associated with RAC production it is often helpful 
to have the number of defects on which the UCR is being written. In the 
revised form of Exhibit 1 this information may or may not be recorded 
in the narrative portions. A block for this data element should be re- 
stored to the top portion of the UCR form. 

Previously generated RACs were able to display repair informa- 
tion for many of the components. The revised form eliminates these 
data elements. If KSC desires to continue repair analysis, resort to 
DRs may be required to obtain the basic information. 

A very helpful data entry omitted in the revised form is the "re- 
lated UCR number(s)." The space allowed for it in the old UCR system 
was inadequate; UCR originators quite often recorded this pertinent in- 
formation in the narrative sections. 

Two entries on Exhibit 1 are of marginal value to purposes of this 
study: category code and manufacturer. Neither is used directly in the 
component analyses. 

Comments on UCR Coding Used in Revised Systems 

One of the continuing difficulties in retrieving UCRs to form 
a data base for analysis has been the inconsistency in key punching part 
numbers. The revised system retains this problem. The retrieval 
program should have the capability of sorting and retrieving by part 
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number, a procedure that is hampered by the varying ways part numbers 
are recorded. 

The revised codes for "functional systems" are an improvement. 
There is now a better agreement between the UCR codes and the codes 
used by DE and the operational personnel for functional systems. 

The revised codes for "defect" while indicating some improve- 
ment from previous codes, still retain a basic problem. At least four 
categories of information are included in the listed codes. These cate- 
gories are: (1) failure mode codes, such as F06, failed closed; (2) 
residual condition codes, such as B07, broken; (3) failure cause codes, 
such as C09, contamination, pollution; and (4) failure symptoms, such 
as I 11, improper output. It is, of course, theoretically possible that 
all four codes could be assigned in a particular instance; current prac- 
tice is to assign only one code. For RAC purposes, both failure mode 
and cause are important. Currently, the analyst must deduce both mode 
and cause from the narrative on the UCR and its associated ICAR. From 
the long range position, coding both the failure mode and cause directly 
on the UCR would facilitate eventual automation of some of the counting 
associated with the RAC tabular displays, a process now requiring hand 
sorting and counting. 

In RAC preparation one of the most important codes in the UCR 
system is the major item. Its importance is simply that UCR retrieval 
is easy using this code, thus making each item on the major item list a 
potential candidate for a RAC. Assembling a data base by retrieving 
UCRs on the basis of part numbers, while more desirable than using 
the major item approach, remains ineffective until a uniform method of 
recording by part numbers is devised and/or until a configuration sys- 
tem is imposed. Elimination of next assembly number has effectively 
closed another approach for assemblying a data base for RAC generation. 
These last two facts make it mandatory to rely on the major item list as 
the primary method for retrieving UCRs for analysis. Revisions to this 
code list should consider this point and items might be included on the 
list which would benefit from RAC generation activities. 
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4. Comments on Proposed Revised UCR Form 

Exhibit 3 is a reproduction of a proposed revised UCR form 
that PRC was requested to review as a part of this study. It differs 
from Exhibit 1 in the following ways: (1) three new data elements are 

incorporated and (2) one data element formerly allotted space for a 
narrative has been allotted a smaller block. The new elements are: 
o Manufacturer' s Part Number: Block 5 

o Serial or Model Number: Block 6 

o Next Assembly Part Number: Block 9 

The data element, "discovered during, " has been relocated from the 
narrative section to the top portion. 

The addition of Serial Number and Next Assembly Part Number 
is strongly endorsed by PRC for reasons discussed earlier. The ad- 
dition of manufacturer part number has no implications for RAC 
generation. 

Although PRC is in concurrence with the revisions as shown in 
Exhibit 3, there are several recommendations discussed in the preced- 
ing paragraphs that should be considered for incorporation before a re- 
vised form is submitted for review. Subsection II. C below collects all 
such recommendations. 

B. Impact of UCR System Revision as of 15 October 1969 on RAC 
Generation Methodology 

The revisions of the UCR system in October 1969 have no effect 
per se on the methodology developed for analyzing UCRs to obtain re- 
liability information for components. There are minor recommenda- 
tions that will be made in the following subsection to improve certain 
aspects of the process, but none arise solely due to incompatibility be- 
tween UCR system and methodology. 

The reduction in UCRs experienced under the revised system is 
not believed to be the direct result of the revised system itself as dis- 
cussed in the preceding subsection. However, even though the observed 
reduction does not impact the methodology directly, there is an effect 
that must be acknowledged. 
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While it is true that the UCR system prior to 15 October 1969 
may not have been responsive to certain DE needs, the UCR originators 
were submitting data sheets on a large variety of problems that were 
amenable to field failure rate analyses. In such analyses all field prob- 
lems or anomalies that cause a work stoppage and/or expenditure of 
monies are of interest. The thrust of field failure rate analysis is to 
isolate those factors influencing the magnitude of the failure rates, take 
steps to remove the influencing factors and thereby reduce the observed 
failure rate. For reasons discussed in Reference 1 it was decided to 
base this work on the UCR system alone (i. e. , excluding DRs). The 
size of the data sample under the earlier (prior to 15 October 1969) 

UCR system was such that it could be considered a representative 
sample of field problems occurring at KSC. If, in fact, the reduction 
of UCRs under the revised system is due to fewer UCRs being generated 
as a result of the insistence that many maintenance -type problems be 
reported on DRs, then the analyses associated with the new system can 
not be said to be based on a representative sample of field failures but 
rather a specific subset of such problems. In any event, the decreased 
rate of UCRs is a serious hindrance to the generation of field failure 
rates. As the decreased rate of UCR submittal is expected to continue 
for the foreseeable future, DE should seriously consider a consolidated 
system combining DRs and UCRs under one reporting system. This 
would enable both maintenance and design engineering information to 
be collected on a single form, and thereby giving much better visibility 
into all KSC GSE problems. The basic methodology for generating field 
failure rates should be directly applicable to DRs as well as to UCRs. 

C. Recommendations 

This section summarizes the various recommendations PRC sub- 
mits for KSC consideration involving various aspects of the UCR system. 

1. UCR Form 

Using Exhibit 3 as a starting point, PRC makes the follow- 
ing recommendations. 
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a. Interchange Blocks 5 and 6 

By placing "serial number" adjacent to "NASA Part 
Number," the originator is more likely to give the NASA serial number 
than the manufacturer's model number. For purposes of this study the 
NASA serial number is preferred over the manufacturer's model number. 

b. Operating Time 

In order not to lose data that may be available, it is 
recommended that Block 16 be arranged in two parts, requesting either 
installed time or operating time. 

c. Number of Defects 

Add a block prior to the narrative section for the re- 
cording of the number of defects being reported on the UCR. This 
block is recommended for clarity; the number of defects is sometimes 
clear in the narrative and sometimes not. 

d. Number of Replacement Parts 

On UCRs reporting a problem on a relatively complex 
equipment, it is of interest to know how many parts were replaced within 
it. Sometimes this information is given in the narrative, sometimes not. 
By including this data element in a block at the top, at least minimal in- 
formation about the unsatisfactory equipment is obtained. 

e. Replacement Part Serial Number 

A block for the serial number of the replacement part 
of the equipment being reported on the UCR is desired. It is used 
as part of the alternate method of computing time for failure rate 
calculations. 

f. Next Assembly Serial Number 

This data item is quite helpful in the analysis of those 
components for which the data base must be retrieved via the "next as- 
sembly part number." A block for this information should be added 
prior to the narrative section. 
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g. Related UCR Numbers 

This information is often available to the UCR origi- 
nator at the time the form is being filled out. These related numbers, 
recorded on the form, are of great benefit to the analyst preparing RACs 
at a later date. It is recommended that a line stating "give related UCR 
numbers; if known" be added in parentheses immediately following the 
title of Block 19, Remarks. 

h. Failure Cause and Mode 

It is recommended that Block 28 be divided into two 
parts (or add a second block altogether), one part for failure mode code 
and the other for failure cause code. For this recommendation to be 
effective, a corresponding revision would be required in the code tables. 

2. Code Tables 

In general, the revisions to the code tables are an improve- 
ment over the earlier versions. The following codes, however, should 
be under continuing surveillance. 

a. . Functional System Codes 

A continuing effort to align the codes of the functional 
systems in a one-to-one correspondence with the functional systems 
names used by other organizations at KSC is encouraged. 

b. Major Item 

Continuing surveillance of the entries on this code 
table is recommended since these items are the most likely candidates 
for RACs. 

c. Defect Code 

It is recommended that this table be revised to show 

codes at least for failure modes and codes for failure causes. This 
recommendation is effective only if implemented in conjunction with 
subsection II. C. 1. h above. 
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3. UCR Printouts 

To obtain a data base of UCRs for RAC generation it is 
necessary that pertinent UCRs be obtainable in hard copies. These 
printouts are used extensively by the analyst; the format of the print- 
outs can materially affect the time required for data analysis. The 
following two recommendations are offered in the interest of ease of 
analysis of the UCRs as they exist after 15 October 1969: (1) space 
between lines or groups of lines rather than single spacing and (2)print 
a UCR and its associated ICAR on one page rather than continuous print- 
ing of the data file. 

4. Augmenting the UCR Data 

It is recommended that the possibility of augmenting the 
UCR system as it now exists with data from the Discrepancy Reports 
be examined for RAC generation purposes. A combined UCR-DR his- 
torical file system is quite likely to give data samples large enough to 
continue the development of field failure rates for KSC use. 

5. UCR Historical File Prior to 15 October 1969 

It is recommended that RACs be generated on the complete 
historical file of UCRs prior to 15 October 1969 as a first priority. 

This would accomplish on a one-time basis the analyses of all data 
available in the historical file and provide all the baseline information 
possible for the proposed KSC handbook.* 

6. Screening of UCRs 

It is recommended that screening of current UCRs be elimi- 
nated so that all UCRs submitted are retained in the historical file. As 
pointed out earlier, all field problems are legitimate data points for 
field failure analyses . If for other DE purposes screening is felt to be 
necessary, consideration should still be given to the retaining of all 
UCRs submitted for purposes of RAC generation. 

*See Volume II, Section III. 
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III. GENERATION OF RACS 


The second task of this study phase was directed toward RAC gen- 
eration based on the UCR historical files. Four subtasks were addressed 
(1) generation of RACs by PRC personnel; (2) transference of the capa- 
bility to produce RACs from PRC personnel to KSC personnel by orga- 
nized training sessions conducted by PRC; (3) minimal supervision of 
KSC generated RACs and incorporation of these RACs together with the 
RACs produced by PRC personnel into a common display system; and 
(4) development of a recommended format for the display of the relia- 
bility information derived by the methodology. This section briefly 
discusses each of these subtasks. 

A. RAC Generation by PRC Personnel 

A list of 11 KSC ground support equipment components selected 
by KSC was supplied to this study. A RAC was requested for each com- 
ponent to be produced in rank order. RACs for the first six were re- 
quired; RACs for as many of the remaining five components as could 
be produced within the study constraints were desired. Modification 
to the study statement of work subsequent to the beginning of this study 
phase enabled completion of all RACs requested by KSC for which there 
was sufficient UCR data. 

The original list of selected components was: 

1. Water System (Water Quench and Industrial Firex Water) 

2. OTY (including RF Instrumentation) 

3. Valves, Solenoid 

4. Holddown Arms (LC-34) 

5. DC Batteries 

6. Circuit Breakers 

7. Relays, family 

8. Compressors 

9. Connectors 

10. Pumps 

11. Tail Service Masts (LC-34) 
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The following were components subsequently added to amplify the second 


component listed above. 

12. RF Carrier Modulator 

MSC-3 9- W 

13. 

Pilot Carrier Generator 

GSC-39-W 

14. 

RF Combining Network 

GSC-39-W 

15. 

Cable Equalizer 

ESC-39-W 

16. 

RF Line Repeater Amplifier 

ASC-39-W 

17. 

RF Line Splitter 

SSC-39-W 

18. 

RF Carrier Demodulator 

DSC-39-W 


Tail Service Masts (LC-34) (number 11 above) were dropped from the 
list due to inability to identify the desired equipment and therefore in 
retrieving the appropriate UCRs. Components numbered 14, 15, and 
17 in the above list contained no UCRs in the historical file and number 
13 had only two UCRs; therefore, no RAC could be produced for these 
four components. 

In addition to the above, a RAC generated in earlier phases of the 
study-effort was updated: Tail Service Masts, Launch Complex 3 9. 

The intent of the update was to explore the effect of the revised UCR 
system on the methodology for RAC generation that had been developed 
in the earlier phases. The results of this review for effect were given 
in Section II of this volume. 

Exhibit 4 lists the RACs produced by PRC during all phases of 
this study. The first 15 were either generated for the first time or 
updated (by KSC direction) during this study phase. The last five, 
produced in an earlier phase, are included in the exhibit since they 
form a part of the recommended handbook, an activity of this phase of 
the study. The exhibit also shows the number of UCRs used to form 
the data base for the reliability analysis for each component. The cur- 
rency date is the date that the UCR historical file was entered to re- 
trieve UCRs. The 20 individual RACs for the components listed in the 
exhibit are a part of Volume II of this report. 
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EXHIBIT 4 - COMPLETED RACS BY COMPONENT NAME, SIZE OF 
ASSOCIATED DATA BASE, AND CURRENCY DATE 



Number of UCRs 
in Data Base 

Old New Total 


RAC 


Currency Date 


Water System 

303 

32 

335 

13 May 1971 

Television System (OTV) 

3,348 

2 

3,350 

26 April 1971 

RF Instrumentation 

39 

0 

39 

26 April 1971 

RF Carrier Demodulator 

167 

0 

167 

26 April 1971 

RF Carrier Modulator 

206 

0 

206 

26 April 1971 

RF Line Repeater Amplifier 

24 

0 

24 

26 April 1971 

Solenoid Valves 

305 

28 

333 

13 May 1971 

Holddown Arms 

25 

9 

34 

13 May 1971 

Batteries 

25 

2 

27 

26 April 1971 

Circuit Breakers 

166 

10 

176 

26 April 1971 

Relays 

260 

34 

294 

26 April 1971 

Compressors 

74 

2 

76 

13 May 1971 

Connectors 

164 

6 

170 

26 April 1971 

Pump Assemblies 

92 

11 

103 

13 May 1971 

Tail Service Masts (LC-39) 

153 

13 

166 

13 May 1971 

Cable Assemblies 

830 

n^, 1 * 

830 

16 June 1969 

Capacitors 

738 

! i 1 

738 

16 June 1969 

Amplifiers 

2,134 

1 17 1 

2,134 

15 Sept. 1969 

Pressure Switches 

120 

Jn* 

120 

16 June 1969 

Regulators 

193 

' \ 
161 
l_ J 

193 

10 April 1968 


Numbers in dotted box are the number of UCRs for the related compo- 
nents in the file after 15 October 1969. The RACs for these components 
do not include these UCRs as they were generated in previous study 
phases. 
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B. Transference of RAC Generation to KSC Personnel 

The ultimate utility of the reliability assessment of components 
is to aid design and reliability engineers at KSC in their on-going sur- 
veillance of the GSC components. As such, the reduction of the UCR 
data should be an on-going task of KSC personnel, generating new RACs 
or updating old ones on a priority basis dictated by the schedules of 
KSC. It has been KSC' s intent from the beginning of these studies to 
develop a methodology for analysis that could become a part of the day- 
to-day KSC activities. This subtask was designed to provide training 
sessions for KSC personnel, conducted by PRC personnel, whereby all 
expertise gained by producing the RACs listed in Exhibit 4 could be 
easily and effectively transmitted. 

Conditions internal to KSC were such that, shortly after contract 
award, it was necessary to curtail the transference activities to the 
preparation of a document giving step-by-step procedures for producing 
a RAC and to the conducting of two training sessions for KSC personnel 
that would be the supervising personnel for any KSC activity in RAC 
preparation. The training document is included as a part of Volume II 
of this report. 

C. Supervision of RAC Generation by KSC Personnel 

Due to the curtailment of KSC activities for production of RACs 
imposed after the study began, no RACs have been produced by KSC 
personnel to date. When this activity is renewed, such RACs may be 
added to those produced in this study in accordance with the instructions 
contained in Volume II of this report. 

D. Recommended Handbook for Displaying KSC Reliability Information 

An important subtask associated with the RACs is to determine 
and recommend a method of displaying the reliability information that 
encourages the use of such data. This subsection discusses the recom- 
mended format. Volume II of this report is the recommended format 
illustrated with all available data. 
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Each RAC is a self-contained report, giving the estimated field 
failure rate (FFR) and associated confidence limits for the component, 
an analysis of the factors that contributed to the magnitude of the FFRs, 
an analysis of failure modes observed, an analysis of failure causes, 
and where possible, an analysis of repair time statistics associated 
with the component's use at KSC. These component reports give de- 
tailed information on the field experience of the component at KSC and 
should be consulted by engineers with an interest in details of the data 
base and analysis results of a given component. 

The recommended handbook format currently includes the RACs 
as an integral part of the handbook. As subsequent RACs are completed, 
the mere volume of paper involved would suggest that RACs should even- 
tually be compiled in separate volumes. 

1. Summarized Baseline UCR Data 

There are expected to be users of the reliability information 
that are not, however, interested in detailed information. For this rea- 
son, the recommended handbook format contains sections of summarized 
data. One such section is called "Baseline UCR Data" and summarizes 
all those data elements derived from the UCR system prior to 15 October 
1969. It is called "baseline" data only because it is more numerous than 
the data elements based on the UCR system subsequent to 15 October 
1969. The Baseline UCR data summaries are tabulations of field failure 
rates and failure classifications recommended for use. These tabula- 
tions are in three major groupings: piece parts, subsystems, and sys- 
tems; the distinction between the levels of these groupings is not strict. 
Piece parts are generally small, relatively high population items found 
in many if not all functional systems. A subsystem, as used in the 
handbook, is generally a collection of piece parts that is still an integ- 
rally functioning unit. A system is generally a collection of subsystems 
and is often not well defined in terms of constituent hardware. 

Included in the tabulations is a confidence factor for each recom- 
mended FFR. The confidence factor in each case is the number of fail- 
ure observations on which the FFR is based. This approach to confidence 
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factors has two advantages. First, a single number can be used which 
is directly indicative of statistical confidence; that is, the higher the 
number of failures the more accurate the indicated FFR. Second, this 
number together with the FFR can be used to enter an exhibit (included 
in Volume II) which provides the upper and lower 90-percent confidence 
limits on the FFR. 

Summarized tabulations for failure causes and failure modes are 
also reported in the section of Baseline UCR data by the three hardware 
levels identified above. A final tabulation summarizes all available re- 
pair time statistics associated with a component in the data base. 

2. Additional UCR Data 

Another section of the recommended format very largely 
parallels the Baseline UCR data section just discussed but is devoted 
entirely to data collected from the UCR system after 15 October 1969. 
As was pointed out in Section II, the FFRs from the old and new sys- 
tems differ considerably in some cases and the new system generally 
contains relatively few UCRs. Therefore, the FFRs, etc. , calculated 
from the old system are taken as the baseline and those derived from 
the new system are presented for whatever influence they might have, 
in the eyes of the individual user, on the baseline figures. Further- 
more, the new figures may be updated as required and should, eventu- 
ally, supplant the baseline figures entirely. No repair data are avail- 
able under the new system nor are any data at all available for some 
items. 

3. Supplementary Data 

It is quite possible that KSC will desire to issue supple- 
mentary data for use until all RACs have been completed. In an ac- 
tually issued handbook this section would contain failure rate data for 
KSC GSE derived from sources other than the UCRs and the RAC 
methodology. In Volume II of this report a section has been reserved 
for this contingency. Contained in that section are recommendations 
for the generation and reporting of such data. 
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4. Methodology for the Reliability Assessment of Components 

It is recommended that this section of the handbook contain 
a reproduction of the previously submitted PRC/SSC report D-1810, 
Reliability Assessment of Components. 30 July 1971, developed for the 
transference subtask above. This has been done in Volume II by making 
only the very minor modifications required to make the document a sec- 
tion of a larger report rather than a report itself. 
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IV. DESIGN CONSIDERATIONS FOR ATTAINING RELIABLE 
REAL-TIME HARD WARE /SO FT WARE CONFIGURATIONS 


This study task addressed the problem of identifying factors that 
affect the reliability of hardware /software configurations for large 
complex, real-time computer systems. This task differs from the 
task reported in Volume II in that it is concerned with ground support 
equipment not yet in design. Therefore, it is primarily concerned with 
identification of methods and procedures that can be used to build reli- 
ability into a system, in particular hardware/ software configurations 
required in the KSC launch support role. The complete results of this 
task are reported in Volume III of this report. This section summarizes 
the results of Volume III. 

A. Purpose and Scope of the Task 

The purposes of this task are to (1) document the collective effort 
to date in the computer industry that has been directed toward improving 
the reliability or quality of hardware/ software configurations; (2) identify 
those factors that affect the combined reliability of such a configuration, 
and (3) develop criteria and/or guidelines useful to KSC in its effort to 
develop and operate such a configuration within limited funds and under 
a severe time constraint. 

The scope of the investigation was limited to the assessment of 
the state-of-the-art in achieving reliable high-quality software for op- 
erational, real-time launch processing systems for space vehicles and 
to the defining of recommendations for further, work in those areas ap- 
pearing to be fruitful and feasible for the accomplishment of the defined 
purpose. 

B. Summary of Volume III Contents 

There are three main discussions contained within Volume III. 

The first addresses aspects of " software reliability." The meaning 
of that phrase is defined for the purposes of this task as "the prob- 
ability that no faults will occur that either delay or abort a scheduled 
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launch that can be attributed to the inability of the software components 
of the hardware /software configuration to perform their intended func- 
tion." Using this as a working definition, a brief investigation was 
made into the relationship or transference value, of proven hardware 
reliability techniques to this concept of "software reliability." As 
shown in Volume III, there are many analogous concepts, terms, and 
techniques used in hardware reliability investigations that have impli- 
cations for software development, implementation, and measurement. 

It is emphasized in the discussion of this topic that a direct transfer- 
ence of all hardware concepts, terms, and techniques is not possible 
nor even desirable. 

The second major discussion of Volume III is devoted to an ex- 
planation of the software production process, including the activities 
and products associated with each stage of the process. Details are 
provided concerning the more effective of the techniques in use today 
in the computer field that have the objective of " building in" software 
reliability into system configurations. 

PRC divided the software production process into the following 
general phases: 

o requirements specification and conceptual analysis 
o requirements allocation and detailed design 
o coding 

o testing 

o integration 

o maintenance ( sustaining engineering) 

Each of the phases of this process is discussed in Volume III according 
to the traditional management trichotomy of planning, implementing and 
measuring. In particular, some of the questions considered are: 

o What means are available for building-in reliability while 
planning each phase of the process? 
o What means are available for use in implementation? 

What techniques are there for measuring the progress of 
each phase? 


o 



PRC R- 1459 
29 


o How can the success of the implementation activity be 
measured? 

o How can we ensure reliability at particular points in the 
process? 

o What data can be gathered in the various phases to promote 
reliability in subsequent phases? 

o What reliability problems are encountered at each phase? 

o Are the identified reliability problems attributable to de- 
cisions made in an earlier phase? 

o What techniques and tools have, in the past, contributed 
to a successful and reliable software system? 

The answers to these questions are collected in Section IV of 
Volume III and provide a set of techniques and tools with which to man- 
age the production of a software system and to build in the needed 
reliability. 

The third major discussion of Volume III addresses the techniques 
which attempt to assess the reliability of software. An evaluation is 
given for further investigations that appear to be fruitful in obtaining 
meaningful software reliability estimates or measures. 

Finally, all recommendations are collected into the last section 
of Volume III and are in the form of criteria and guidelines that are in- 
tended to enhance the chance of obtaining a reliable system under severe 
cost and time constraints. 

The appendix of Volume III contains an annotated bibliography of 
various aspects of the problems considered in this task. 
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grams," NASA, George C. Marshall Space Flight Center, NASA 
TM X-64574, 8 March 1971. 
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(KSC Form 22-296, 1/70), prepared by FEC R &QE Branch. 

6. Skylab System Review Schedule; one- sheet review schedule. 

7. Handwritten list of mechanical component problems prepared 
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8. "Nonconformance Reporting andCorrective Action System," 

John F. Kennedy Space Center Management Instruction, KMI 
5310, 11A/QA, 8 January 1971. 

9. Four pages extracted from Boeing Company Report, Number 
D5-16739-3, pages 6-12 through 6-15; data sheets for three part 
numbers within the Tail Service Mast System. 

10. Revised UCR ADP Code Tables, dated 27 March 1970 and 2 March 
1971. 

11. Master List - Old System, UCR Code Tables dated October 1969. 

12. "Saturn V Composite Mechanical Schematic," George C. Marshall 
Space Flight Center, IOM 30531 REV P, 31 August 1970. 

13. "Methods and Guidelines to Reduce Launch Cost, Executive Sum- 
mary," Martin Marietta, 1 September 1969. 

"Methods and Guidelines to Reduce Launch Cost, Study Summary," 
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Memorandum transmitting proposed outline for a Handbook on 
Redundancy and Redundancy Verification, W. R. McMurran, 

NASA, John F. Kennedy Space Center, 15 April 1971. 

Phase I Progress Report for Study of Techniques for Automatic 
Self-Contained Readiness Assessment and Fault Isolation for 
Ground and On-Board Mechanical Systems, General Electric 
Company, 70-831-892, 20 July 1970. 

Phase III Progress Report for Study of Techniques for Automatic 
Self-Contained Readiness Assessment and Fault Isolation for 
Ground and On-Board Mechanical Systems, General Electric 
Company, 70-831-894, 19 October 1970. 

Final Report for Study of Techniques for Automatic Self-Contained 
Readiness Assessment and Fault Isolation for Ground and On-Board 
Mechanical Systems, General Electric Company, 70-831-895, 

13 November 1970. 

Unsatisfactory Condition Report, KSC Form 14-14A, (9/69). 

Investigation and Corrective Action Report, KSC Form 14-14B 
(Rev. 12/69). 

"Space Shuttle Technology Conference," Volume 1 - Operations, 
Maintenance and Safety, NASA, John F. Kennedy Space Center, 
TR-1113, 3 May 1971; Volume 2 - Biotechnology, TR-1114, 

3 May 1971. 

LC-34/37 Mechanical Equipment for LC-39, listing dated 
11 December 1970. 

Systems Engineering Study of LC-34/37 Reusable Equipment, 
internal KSC report, Serial No. SED-3-ER-04, 29 June 1970. 

Tail Service Mast, System 008, Design Changes, listing, no date. 

09, LUT Mechanical Design Changes, Structural, listing, no date. 

Model numbers for the LC-39 OTV-RF cable transmission system; 
letter communication, 10 June 1971. 

Complete UCR printout for old system for following categories: 

o Entire file, run dated 26 April 1971. 
o Functional system, Industrial Water Supply, run dated 
26 April 1971. 

o Functional system, RF Instrumentation, run dated 26 April 1971. 

o Functional system, Hold Down Arms, run dated 26 April 1971. 

o Functional system, Water Quench, run dated 26 April 1971. 

o Functional system. Television, run dated 26 April 1971. 



Selected UCR elements printout for old system for following 
categories: 

o By Part Number, 10 UCR elements, run dated 19 March 
1971. 

o By Functional System, 7 UCR elements, run dated 26 April 
1971. 

o By Functional System, 7 UCR elements, run dated 18 May 
1971. 

o By Major Item, 10 UCR elements, run dated 26 April 1971. 

o By KSC Report Number, 8 UCR elements, run dated 18 May 

1971. 

Complete UCR printout including complete printout of ICAR's for 
new system for following categories: 

o Entire file, run dated 13 May 1971. 

o Functional System, Swing Arms -General, run dated 19 May 
1971. 

o Functional System, Hold Down Arms and Liftoff, run dated 
19 May 1971. 

o Functional System, Tail Service Masts, run dated 19 Mav 
1971. 

Selected UCR elements printout for new system for following 
categories: 

o By major item, 8 UCR elements, run dated 21 April 1971. 

o By major item, 7 UCR elements, run dated 13 May 1971. 

o By location, 8 UCR elements, run dated 21 April 1971. 

o By functional system, 6 UCR elements, run dated 10 May 

1971. 

o By part number, 7 UCR elements, run dated 13 May 1971. 
o By KSC report number, 6 UCR elements, run dated 13 May 
1971. 

ICARs for old system, complete printout for entire file, run dated 
27 March 1971 . 

Thirty-nine UCRs screened from the new UCR program; received 
September 1972. 

Draft copy of a KSC specification, received December 1971. 

One blank copy and one sample copy of a Program Trouble Report, 
KSC form 23-225 Rev. 9/66; received January 1972. 

Sample copies of failure rate handbooks; received February 1972, 
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